Learning and Recognition of Complex Human Actions

Executive Summary This deliverable concerns the third objective of WP5, which is to develop a learning and recognition algorithm for complex actions. We have extended the factored-HMM representation of manipulation actions developed in Year 2 by another layer of dynamic models of goals, in which details of manipulation movements are abstracted away. The focus of work in Year 3 is to model the dynamics of human intention that results in the observed human complex behaviour. This report is a revised and extended version of the report submitted in Year 2, and contains new analysis results. • Two key capabilities: We identify two key capabilities for a system to understand a complex human behaviour. The first capability is to learn normal behaviour that allows induction of the agent's goals from partial or noisy observation. The second is to be able to attribute any deviations from normal behaviour to the background knowledge (hidden) or other differentiating scene features (observable). Achieving these capabilities requires a representation that not only encodes the dynamics of the scene, but also allows information about other agents' beliefs, plans and intentions to be integrated. Human complex behaviour is represented by a hierarchical, factorized hidden Markov models (factored-HMMs) as shown in Figure 1. It consists of two levels of HMMs. The nodes of HMM at the lower level are factorized into manipulation types (h) and object states (o) which produces observable visual features f h and f o , respectively. The links between nodes represent the dependencies in human decision process. At time t, hand movement h t depends on o t−1 and g t , which models the goal of the acting agent. Complex manipulation is considered to be caused by the agent's evolving manipulation goals, which are modeled by another layer of HMMs. To understand complex behaviour, the observer needs to represent the agents' prior knowledge about goals and how they may evolve. • Modeling evolving goals: We suggest two candidate models based on goal switching and goal composition. " Goal switching " is a process whereby one manipulation action is switched into another manipulation action, and can be used to account for sequences of actions. " Complex goal composition " is a process whereby a goal is augmented by sub-goals to account for deviations from normal behaviour. Factored-HMMs have been extended to take into account evolving goal models, and used to segment input video streams into …